Downsample zarr

Demonstrates downsampling zarr using xarray and dask with data from SMAP
Author

Aimee Barciauskas

Published

January 3, 2023

Run this notebook

You can launch this notebook in VEDA JupyterHub by clicking the link below.

Launch in VEDA JupyterHub (requires access)

Learn more

Inside the Hub

This notebook was written on the VEDA JupyterHub and as such is designed to be run on a jupyterhub which is associated with an AWS IAM role which has been granted permissions to the VEDA data store via its bucket policy. The instance used provided 16GB of RAM.

See (VEDA Analytics JupyterHub Access)[https://nasa-impact.github.io/veda-docs/veda-jh-access.html] for information about how to gain access.

Outside the Hub

The data is in a protected bucket. Please request access by emailng aimee@developmentseed.org or alexandra@developmentseed.org and providing your affiliation, interest in or expected use of the dataset and an AWS IAM role or user Amazon Resource Name (ARN). The team will help you configure the cognito client.

You should then run:

%run -i 'cognito_login.py'

Approach

This notebook demonstrates 2 strategies for to subselect data from a Zarr dataset in order to visualize using the memory of a notebook.

  1. Downsample the temporal resolution of the data using xarray.DataArray.resample
  2. Coarsening the spatial aspect of the data using xarray.DataArray.coarsen

A strategy for visualizing any large amount of data is Datashader which bins data into a fixed 2-D array. The call to rasterize ensures the use of the datashader library to bin the data.

About the data

The SMAP mission is an orbiting observatory that measures the amount of water in the surface soil everywhere on Earth.

Load libraries

import s3fs
import xarray as xr
import hvplot.xarray
import geoviews as gv
import datashader as dsh
from holoviews.operation.datashader import rasterize

gv.output(size=300)

Optional: Create and Scale a Dask Cluster

We create a separate Dask cluster to speed up reprojecting the data (and other potential computations which could be required and are parallelizable).

Note if you skip this cell you will still be using Dask, you’ll just be using the machine where you are running this notebook.

from dask_gateway import GatewayCluster, Gateway

gateway = Gateway()
clusters = gateway.list_clusters()

# connect to an existing cluster - this is useful when the kernel shutdown in the middle of an interactive session
if clusters:
    cluster = gateway.connect(clusters[0].name)
else:
    cluster = GatewayCluster(shutdown_on_close=True)

cluster.scale(16)
client = cluster.get_client()
client

Open the dataset from S3

s3 = s3fs.S3FileSystem()
root = "veda-data-store-staging/EIS/zarr/SPL3SMP.zarr"
store = s3fs.S3Map(root=root, s3=s3)
ds = xr.open_zarr(store=store)
ds
<xarray.Dataset>
Dimensions:                        (northing_m: 406, easting_m: 964,
                                    datetime: 1679)
Coordinates:
  * datetime                       (datetime) datetime64[ns] 2018-01-01 ... 2...
  * easting_m                      (easting_m) float64 -1.735e+07 ... 1.735e+07
  * northing_m                     (northing_m) float64 7.297e+06 ... -7.297e+06
Data variables: (12/26)
    albedo                         (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    albedo_pm                      (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    bulk_density                   (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    bulk_density_pm                (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    clay_fraction                  (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    clay_fraction_pm               (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    ...                             ...
    static_water_body_fraction     (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    static_water_body_fraction_pm  (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    surface_flag                   (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    surface_flag_pm                (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    surface_temperature            (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>
    surface_temperature_pm         (northing_m, easting_m, datetime) float32 dask.array<chunksize=(100, 100, 100), meta=np.ndarray>

    • Select the variable of interest (soil moisture for this example).

    soil_moisture = ds.soil_moisture
soil_moisture
    <xarray.DataArray 'soil_moisture' (northing_m: 406, easting_m: 964,
                                   datetime: 1679)>
dask.array<open_dataset-e9352a6cb9ac62da20f04f798593159fsoil_moisture, shape=(406, 964, 1679), dtype=float32, chunksize=(100, 100, 100), chunktype=numpy.ndarray>
Coordinates:
  * datetime    (datetime) datetime64[ns] 2018-01-01 2018-01-02 ... 2022-09-09
  * easting_m   (easting_m) float64 -1.735e+07 -1.731e+07 ... 1.735e+07
  * northing_m  (northing_m) float64 7.297e+06 7.26e+06 ... -7.26e+06 -7.297e+06
Attributes:
    coordinates:  /Soil_Moisture_Retrieval_Data_AM/latitude /Soil_Moisture_Re...
    long_name:    Representative DCA soil moisture measurement for the Earth ...
    units:        cm**3/cm**3
    valid_max:    0.5
    valid_min:    0.019999999552965164

    Strategy 1: Downsample the temporal resolution of the data

    To plot one day from every month, resample the data to 1 observation a month.

    somo_one_month = soil_moisture.resample(datetime="1M").nearest()

    Quick plot

    We can generate a quick plot using hvplot and datashader.

    # workaround to avoid warnings that are triggered within Dask.
import warnings

warnings.filterwarnings(
    "ignore", message="All-NaN slice encountered", category=RuntimeWarning
)
    somo_one_month.hvplot(
    x="easting_m",
    y="northing_m",
    groupby="datetime",
    crs="epsg:6933",
    coastline=True,
    rasterize=True,
    aggregator="mean",
    frame_height=150,
    widget_location="bottom",
)
    WARNING:param.main: Calling the .opts method with options broken down by options group (i.e. separate plot, style and norm groups) is deprecated. Use the .options method converting to the simplified format instead or use hv.opts.apply_groups for backward compatibility.

    Reproject before plotting

    Reproject the data for map visualization.

    somo_one_month = somo_one_month.transpose("datetime", "northing_m", "easting_m")
somo_one_month = somo_one_month.rio.set_spatial_dims(
    x_dim="easting_m", y_dim="northing_m"
)
somo_one_month = somo_one_month.rio.write_crs("epsg:6933")
somo_reprojected = somo_one_month.rio.reproject("EPSG:4326")
somo_reprojected
    <xarray.DataArray 'soil_moisture' (datetime: 57, y: 1046, x: 2214)>
array([[[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
...
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)
Coordinates:
  * x            (x) float64 -179.9 -179.8 -179.6 -179.4 ... 179.6 179.8 179.9
  * y            (y) float64 84.96 84.8 84.64 84.48 ... -84.64 -84.8 -84.96
  * datetime     (datetime) datetime64[ns] 2018-01-31 2018-02-28 ... 2022-09-30
    spatial_ref  int64 0
Attributes:
    coordinates:  /Soil_Moisture_Retrieval_Data_AM/latitude /Soil_Moisture_Re...
    long_name:    Representative DCA soil moisture measurement for the Earth ...
    units:        cm**3/cm**3
    valid_max:    0.5
    valid_min:    0.019999999552965164

    Note that this is now a fully materialized data array - when we reproject we trigger an implicit compute.

    Create a geoviews dataset and visualize the data on a map.

    kdims = ["datetime", "x", "y"]
vdims = ["soil_moisture"]
xr_dataset = gv.Dataset(somo_reprojected, kdims=kdims, vdims=vdims)
images = xr_dataset.to(gv.Image, ["x", "y"])

rasterize(
    images, precompute=True, aggregator=dsh.mean("soil_moisture")
) * gv.feature.coastline

    Strategy 2: Coarsen spatial resolution of the data

    Below, we coarsen the spatial resolution of the data by a factor of 4 in the x and 2 in the y. These values were chosen because they can be used with the exact boundary argument as the dimensions size is a multiple of these values.

    You can also coarsen by datetime, using the same strategy as below but replacing easting_m and northing_m with datetime. If {datetime: n} is the value give to the dim argument, this would create a mean of the soil moisture average for n days.

    Once the data has been coarsned, again it is reprojected for map visualization and then visualized using Geoviews.

    coarsened = soil_moisture.coarsen(dim={"easting_m": 4, "northing_m": 2}).mean()

coarsened = coarsened.transpose("datetime", "northing_m", "easting_m")
coarsened = coarsened.rio.set_spatial_dims(x_dim="easting_m", y_dim="northing_m")
coarsened = coarsened.rio.write_crs("epsg:6933")
coarsened_reprojected = coarsened.rio.reproject("EPSG:4326")
coarsened_reprojected
    <xarray.DataArray 'soil_moisture' (datetime: 1679, y: 315, x: 667)>
array([[[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
...
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)
Coordinates:
  * x            (x) float64 -179.7 -179.2 -178.7 -178.1 ... 178.6 179.2 179.7
  * y            (y) float64 84.77 84.23 83.7 83.16 ... -83.64 -84.18 -84.72
  * datetime     (datetime) datetime64[ns] 2018-01-01 2018-01-02 ... 2022-09-09
    spatial_ref  int64 0
Attributes:
    coordinates:  /Soil_Moisture_Retrieval_Data_AM/latitude /Soil_Moisture_Re...
    long_name:    Representative DCA soil moisture measurement for the Earth ...
    units:        cm**3/cm**3
    valid_max:    0.5
    valid_min:    0.019999999552965164
    _FillValue:   3.402823466e+38
    kdims = ["datetime", "x", "y"]
vdims = ["soil_moisture"]
xr_dataset = gv.Dataset(coarsened_reprojected, kdims=kdims, vdims=vdims)
images = xr_dataset.to(gv.Image, ["x", "y"])
    rasterize(
    images, precompute=True, aggregator=dsh.mean("soil_moisture")
) * gv.feature.coastline

    Cleanup

    When using a remote Dask cluster it is recommented to explicitly close the cluster.

    client.shutdown()